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ABSTRACT 


This  piper  compares  Wolfe's  quadratic  programming  algorithm  with 
Cottle  and  Dantzig's  principle  pivot  method.  It  is  shown  that  Wolfe's 
algorithm  requires  more  operations. 
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In  this  paper  we  will  compare  Wolfe's  quadratic  programming 
algorithm  with  the  principle  pivot  method  proposed  by  Cottle  and 
Dantzig  [3].  In  the  first  section  we  describe  the  problem  and  define 
some  notation.  In  sections  2  and  3  the  algorithms  are  briefly  described 
and  examples  worked  out.  Then  in  section  4  we  discuss  the  merits  of 
each  algorithm. 

No  typographical  distinction  is  made  between  vectors  and  scalars. 
Matrices  are  designated  by  capital  letters.  The  transpose  of  w  is 
written  w’ .  All  vectors  are  column  vectors. 

1.  Problem  Formulation  The  problem  we  wish  to  consider  may  be  formulated 
as  follows.  Find  x  such  that 

f(x)  =  “*£(c'x+  \  x'Dx)  (1-1) 

where  R  =  {xeEn;  Ax  b,  x  >.  0}, 
ceEn,  bcEm 

A  is  a  mxn  matrix  and  D  is  a  nxn  matrix. 

We  can  assume  without  loss  of  generality  that  D  is  a  symmetric  matrix. 

If  the  region  R  is  empty,  then  the  problem  (1.1)  has  no  solution. 

If  R  contains  a  single  point,  then  the  problem  has  a  trivial,  unique 
solution.  We  will,  therefore,  assume  that  the  interior  of  the  region  R 
is  non-empty. 

Define  the  Lagrange  function 

L(x,  y,  v)  =  c'x  +  —  x'Dx  -  y'(Ax  -  b)  -  v'x  (1.2) 

.  ..in  „n 

where  yeE  ,  vcE  . 

Under  the  assumptions  made  on  R  we  know  that  the  Kuhn-Tucker  [5]  conditions 
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are  necessary.  That  Is,  x  exist  only  if  there  exist  vectors  y  and  v  such 
that  x,  y,  v  satisfy 


c'  +  x'D  -  y'A  -v'  -  0 
Ax  -  b  >  0 

y' (Ax  -  b)  -  0,  v'x  -  0 
x  >  0,  y  >  0,  v  >  0. 


*\ 


f  (1.3) 


J 


Solving  these  equations, el Inina  ting  v,  rearranging  and  recalling  that 
D'  -  D  we  have 


c  +  Dtx  -  A'y  >  0 
-b  +  Ax  >_  0 

y'(-b  +  Ax)  -  0,  x'(c  +  Dx  -  A'y)  -  0 
x  >0,  y  >  0. 


•\ 


>  (1.*) 


J 


We  can  express  these  conditions  In  a  nore  conpact  way.  Put 


z 


q 


(1.5) 


Then  (1.4)  Is  equivalent  to  what  Cottle  and  Dantzlg  [3]  call  the  Fundamental 
Problem. 


v  ■  q  +  Mz 
w'z  ■  0 
v  >  0,  i  >  0 


>  (1.6) 


where  M  la  a  (n+m)x(n+m)  matrix, 

.  -trtm 

and  w,  q,  z  cE 


We  shall  be  concerned  with  two  different  algorithms  which  can  be  used  to 
solve  the  Kuhn-Tucker  conditions  as  expressed  In  (1.6). 


2.  Wolfe's  Algorithm  The  set  of  equations  in  (1.6)  is  linear  except 
for  w'z  =  0.  Wolfe's  algorithm  is  a  modification  of  linear  programming 
algorithms  to  account  for  the  equation  w'z  =  0. 

Make  b  >_  0  in  (1.4)  and  subtract  and  add  the  slack  and  surplus 
variables  tj  such  that  t  >  0,  teE11.  Let  G  be  the  matrix  [+6^,.]  of  coef¬ 
ficients  of  t.  Now  define  the  surplus  vector  s  >.  0,  seEn  and  rewrite 
(1.4)  as 


c  +  Dx  -  A'y  -  s  =  0 
-b  +  Ax  -  Gt  =  0 

y'(-b  +  Ax)  =0,  x'(c  +  Dx  -  A'y)  =  0 
x,  s  >  0  y,  t  >  0. 


>  (2.1) 

> 


Or  equivalently. 


(x*  ,  y,)  J  “  0 


[xi  y.  s,  t]  >  o. 


>  (2.2) 


j 


J 


Now  using  the  definitions  in  (1.5)  and  putting  w  =  [s,  Gt]  we  have 


q  -  (m,  -i)[zj  =  o 


w'z  ■  0 
v  >,  0,  z  >_  0 


\  (2.3) 


which  is  of  course  (1.6)  reformatted. 


Now  consider  the  problem 
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n 

min  E  u 
i=l 


subject  to 


(M,  -  I,  I)  pH 
W 

Lu  J 


w  >,  0,  z  >  0,  u  >  0. 


>  (2.4) 


If  the  solution  to  (2.4)  has  u  ■  0  and  at  the  same  time  we  have  w'z  =  0, 
then  we  have  solved  (1.4)  and  hence  (1.1)  our  original  problem. 

Wolfe's  algorithm  is  to  solve  (2.4)  using  a  linear  programming 
algorithm  modified  to  ensure  that  w'z  =  0  at  all  times.  We  can  hold 
w'z  =  0  if  at  each  iteration  we  ensure  that  if  one  of  the  vectors  asso¬ 
ciated  with  w^  or  z^  is  in  the  basis  then  the  other  is  not. 

The  condition  w'z  =  0  guarantees  that  no  more  than  (n+ra)  variables  in 
(2.3)  need  be  non-zero.  Hence,  the  solution  to  (2.3)  is  one  of  the  basic 
solutions  to  (2.4).  In  fact  it  can  be  shown  that  in  spite  of  the  require¬ 
ment  that  w'z  =  0,  the  linear  programming  algorithms  still  converge.  (For 
example,  see  Hadley  [4]  pages  214-221.) 

Wolfe's  algorithm  requires  that  every  iteration  maintain  w'z  =  0, 
hence  we  must  start  with  a  basic  feasible  solution  for  which  this  is  true. 
Since  q  has  (n+m)  components,  if  we  put 

c  >  0  i  “  1,  . . . ,  n 

otherwise, 

c<0  i  ■  1 . n 

otherwise 
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and 

Gt°  =  -  b. 

Then  [z°,  w°,  u°]  is  an  initial  basic  feasible  solution  to  (2.4)  which 
satisfies  (2.3). 

Convergence  is  guaranteed  (assuming  no  degeneracy)  if  D  is  positive 
definite.  Hadley  [4]  describes  a  method  due  to  Charnes  [1]  for  solving 
the  semidefinite  case.  Instead  of  using  D  we  use  (D  +  El)  for  e  >  0  and 
arbitrarily  small.  Then  (D  +  el)  is  positive  definite  and  we  have 
convergence. 

We  will  now  consider  two  examples.  The  first  is  positive  definite 
and  the  second  is  positive  semidefinite.  However,  both  problems  have  unique 
solutions . 

The  first  problem  we  wish  to  consider  is 
min  x|  +  ~  8x^  -  10x2* 

subject  to 

3x^  +  2x£  — 
xl»  x2  L  °* 

For  this  problem  we  have 

C  "  [-10  ]  »  D  =  [o  2]  *  A  =  (“3»  “  2)»  b  =  (~6)* 

And  Wolfe's  algorithm  gives  us  (from  2.4) 
min  +  Uj 

subject  to 


'  2 

0 

3 

-1 

0 

0 

1 

0 

z 

'8  ' 

0 

2 

2 

0 

-1 

0 

0 

1 

w 

= 

10 

.-3 

-2 

0 

0 

0 

-1 

0 

0. 

-U. 

.-6. 
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with  an  initial  basic  feasible  solution 
[z°,  w°,  u°]  ■=  [0,  0,  0,  0,  0,  6,  8,  10]. 

Tableaux  I  through  IV  are  for  the  above  problem.  We  see  in  tableaux 
I  and  III  that  we  cannot  apply  the  normal  simplex  rules  for  exchanges  of 
basis  vectors  and  Wolfe's  modification  comes  into  play.  The  optimal 
solution  to  this  problem  is  [z,  w,  u]=[4/13,  33/13,  32/13,  0,  0,  0,  0,  0]. 


TABLEAU  I:  INITIAL  TABLEAU 


Z1 

z2 

Z3 

W1 

w2 

W3 

U1 

U2 

2 

0 

3 

-1 

0 

0 

1 

0 

8 

U1 

0 

2 

2 

0 

-1 

0 

0 

1 

10 

u2 

© 

-2 

0 

0 

0 

-0 

0 

0 

-6 

w3 

2 

2 

5 

-1 

-1 

0 

0 

0 

18 

TABLEAU  II 

0 

-4/3 

© 

-1 

0 

-2/3 

1 

0 

4 

U1 

0 

2 

2 

0 

-1 

0 

0 

1 

10 

U2 

1 

2/3 

0 

0 

0 

1/3 

0 

0 

2 

Z1 

0 

2/3 

5 

-1 

-1 

-2/3 

0 

0 

14 

TABLEAU  III 

0 

-4/9 

1 

-1/3 

0 

-2/9 

1/3 

0 

4/3 

Z3 

0 

(26/9) 

0 

2/3 

-1 

4/9 

-2/3 

1 

22/3 

U2 

1 

2/3 

0 

0 

0 

1/3 

0 

0 

2 

Z1 

0 

26/9 

0 

2/3 

-1 

4/9 

-5/3 

0 

22/3 
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TABLEAU  IV 

0 

0 

1 

-3/13 

-2/13 

-2/13 

3/13 

2/13 

32/13 

0 

1 

0 

3/13 

-9/26 

2/13  - 

-3/13 

9/26 

33/13 

1 

0 

0 

-2/13 

-3/13 

3/13 

2/13 

-3/13 

4/13 

0 

0 

0 

0 

0 

0 

-1 

-1 

0 

Our  second  example  Is 
2 

min  2x2  -  2x^  -  3x2 
subject  to 

x1  +  4x2  £  4 
+  x2  =  2  * 

For  this  problem  we  have 

••[:!]'  "■[::]■*•[:::;]•  “G]. 

Clearly,  D  is  semidefinite.  Our  initial  basic  feasible  solution  is 
[ z° ,  w° ,  u°]  =  [0,  0,  0,  0,  0,  0,  4,  2,  2,  3].  And  the  optimal  solution  is 

[z,  w,  u]  =  [7/4,  1/4,  0,  2,  0,  0,  5/4,  0,  0,  0]. 

This  problem  takes  five  iterations  (Tableaux  V  thru  X).  Note  that 
in  Tableau  VI  we  cannot  enter  z^  or  z^  into  the  basis  so  that  z^  must 
enter  the  basis  even  though  the  last  row  is  0. 
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TABLEAU  V 


Z1 

Z2 

Z3 

Z4 

W1 

W2 

W3 

w4 

U1 

U2 

0 

0  . 

1 

1 

-1 

0 

0 

0 

1 

0 

2  U1 

0 

o 

4 

1 

0 

-1 

0 

0 

0 

1 

3  u2 

-1 

-4 

0 

0 

0 

0 

-1 

0 

0 

0 

-4  w3 

-1 

-1 

0 

0 

0 

0 

0 

-1 

0 

0 

-2  w4 

0 

4 

5 

2 

-1 

-1 

0 

0 

0 

0 

5 

TABLEAU  VI 

0 

0 

1 

1 

-1 

0 

0 

0 

1 

0 

2ul 

0 

1 

1 

1/4 

0 

-1/4 

0 

0 

0 

1/4 

3/4  z2 

9 

0 

4 

1 

0 

-1 

-1 

0 

0 

1 

-1  w3 

-1 

0 

1 

1/4 

0 

-1/4 

0 

-1 

0 

1/4 

.-5/4  w4 

0 

0 

1 

1 

-1 

0 

0 

0 

0 

-1 

2 

TABLEAU  VII 

0 

0 

1 

1 

-1 

0 

0 

0 

1 

0 

2 

0 

1 

1 

1/4 

0 

-1/4 

0 

0 

0 

1/4 

3/4  z2 

1 

0 

-4 

-1 

0 

1 

1 

0 

0 

-  -1 

1  Z1 

0 

0 

© 

-3/4 

0 

3/4 

1 

-1 

0 

-3/4 

-1/4  w4 

0 

0 

1 

1 

-1 

0 

0 

0 

0 

-1 

2 

TABLEAU  VIII 

0 

0 

0 

3/4 

-1 

1/4 

1/3  - 

■1/3 

1 

-1/4 

23/12  U;L 

0 

1 

0 

0 

0 

0 

1/3  - 

■1/3 

0 

0 

8/12  z.( 

4. 

1 

0 

0 

0 

0 

0 

-1/3 

4/3 

0 

0 

4/3  z ! 

0 

0 

1 

© 

0 

-1/4 

-1/3 

1/3 

0 

1/4 

1/12  z3 

0 

0 

0 

3/4 

-1 

1/4 

1/3  - 

■1/3 

0 

-1/4  23/12 
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TABLEAU  IX 


Z1 

*2 

z3 

*4 

W1 

w2 

w3 

w4 

U1 

U2 

0 

0 

-3 

0 

-1 

1 

© 

-4/3 

1 

-1 

5/3  u 

0 

1 

0 

0 

0 

0 

1/3 

-1/3 

0 

0 

8/12  z 

1 

0 

0 

0 

0 

0 

-1/3 

4/3 

0 

0 

4/3  z 

0 

0 

4 

1 

0 

-1 

-4/3 

4/3 

0 

1 

1/3  z 

0 

0 

-3 

0 

-1 

1 

4/3 

-4/3 

0 

-1 

5/3 

TABLEAU 

X 

0 

0 

-9/4 

0 

-3/4 

3/4 

1 

-1 

3/4 

-3/4 

5/4  w 

0 

1 

3/4 

0 

1/4 

-1/4 

0 

0 

-1/4 

1/4 

1/4  z 

1 

0 

-3/4 

0 

-1/4 

1/4 

0 

1 

1/4 

-1/4 

7/4  z 

0 

0 

1 

1 

-1 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

-1 

-1 

0 
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3.  The  Principle  Pivot  Method  For  convenience  we  repeat  the  Fundamental 
Problem.  Find  vectors  w,  z,  such  that 


w  ••  q  +  Mz, 
w'z  =  0, 
z  >  0,  w  >  0 


r  (3.D 


nlm 

where  M  is  a  (n  +  m)  square  matrix  and  w,  q,  z  e  E  .  We  have  seen 
that  this  system  (3.1)  is  equivalent  to  the  Kuhn-Tucker  conditions  (1.4) 
and  has  a  unique  solution  if  D  is  positive  definite. 

From  (3.1)  we  have  w'z  =  wizi  =  ®  which  implies  that  each  non¬ 

negative  pair  (w^,  z^)  has  at  least  one  zero  component.  We  call  (w^,  z i) 
the  complementary  pair. 

A  pivot  is  an  operation  on  the  equation  w  =  q  +  Mz  which  exchanges 

the  roles  of  w^  and  z^  for  some  j  and  k.  We  call  (w^ ,  z^)  the  pivotal  pair. 

The  pivot  is  accomplished  by  solving  one  of  the  (n+m)  equations  for  z^  in 

terms  of  z . ,  i^k  and  w . .  We  then  eliminate  the  variable  z.  from  the 
i  j  k 

other  (n+m)  -  1  equations.  The  result  is  a  new  system 


V  V  V  V 

w  *>  q  +  M  z 


(3.2) 


in  which  the  roles  of  the  pivotal  pair  are  interchanged,  i.e.,  Wj 

appears  on  the  right  and  zk  appears  on  the  left  in  (3.2).  The  operation 

is  not  defined  if  m„  =  0.  If  in  the  pivotal  pair,  j  =  k,  then  it  is 

J 

called  a  complementary  pivot.  A  sequence  of  complementary  pivots  is  a 
principle  pivot.  The  reader  will  recognize  the  pivot  operation  as  an 
iteration  of  the  classic  Gauss-Jordan  reduction  technique  for  solving 
linear  equations. 

A  basic  solution  to  (3.2)  is  a  solution  with  no  more  than  (n+m)  non¬ 
zero  variables.  A  basic  solution  to  (3.2)  which  also  satisfies  (3.1)  is 
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a  complementary  basic  solution. 

Cottle  and  Dantzig  [3]  show  that,  under  certain  assumptions  on 
the  matrix  M,  a  sequence  of  principle  pivots  on  (3.2)  starting  from  a 
basic  solution  to  (3.2)  will  converge  to  a  complementary  basic  solution 
to  (3.1)  in  a  finite  number  of  iterations  (assuming  no  degeneracy).  This 
is  the  Principle  Pivot  Method.  Convergence  of  this  algorithm  is  proved 
in  the  case  where  M  has  positive  principle  minors  [3].  If  M  is  positive 
definite,  then  it  has  positive  principle  minors.  In  the  case  where  M 
is  positive  semidefinite,  some  of  the  principle  minors  are  zero.  Cottle 
[2]  has  provided  a  modification  of  the  principle  pivot  algorithm  which 
converges  if  M  is  positive  semidefinite  to  a  solution  or  an  indication 
of  an  unbounded  solution.  The  proofs  for  convergence  depend  on  the  ob¬ 
servations  that  the  properties  of  "positive  principle  minors"  and 
"positive  (semi) definite"  remain  invariant  under  the  operation  of  principle 
pivot. 

To  translate  the  requirements  on  M  to  equivalent  requirements  on  D 
we  observe  that  if  D  is  positive  definite  or  positive  semidefinite,  then 
M  is  positive  semidefinite  since 


z'Mz 


(x'»  y' ) 


=  x'Dx  +  (y'Ax  -  x'A'y) 


=  x'Dx  . 


Hence,  if  D  is  positive  (semi)definite  we  know  that  the  algorithm  will 
converge  or  give  an  indication  of  an  unbounded  solution. 

We  now  consider  the  examples  solved  by  Wolfe's  method  in  section  2. 
We  will  use  the  tableau  suggested  by  Cottle  [2].  The  first  problem  is 
in  tableaux  XI  through  XIV  and  the  second  problem  in  tableaux  XV  through 


XVIII. 
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TABLEAU  XI 

TABLEAU  XII 

Z1  Z2 

Z3 

W3 

Z2  Z3 

W1 

-8 

2  0 

3 

W1 

-4 

0  - 

4/3  Q 

W2 

-10 

0  2 

2 

W2 

-10 

0 

2  2 

W3 

6 

©  '2 

0 

Z1 

2 

-0  - 

2/3  0 

TABLEAU  XIII 

TABLEAU  XIV 

W3  Z2 

W1 

W3 

W2  V1 

Z3 

4/3 

0  4/9 

-1 

•  Z3 

32/13 

0 

0  -1 

W2 

-22/3 

o  (26/9) 

0 

Z2 

33/13 

0 

-1  0 

Z1 

2 

-1  -2/3 

0 

Z1 

4/13 

-1 

0  0 

TABLEAU  XV 

TABLEAU  XVI 

W1 

-2 

Z1  Z2  Z3 

0  0  1 

Z4 

1 

W1 

-2 

W4  Z2 

0  0 

Z3  Z4 
>© 

W2 

-3 

0  4  4 

1 

w2 

-3 

0  4 

4  1 

W3 

4  • 

-1  -4  0 

0 

W3 

2 

0  -3 

0  0 

W4 

2  0-1  0 

0 

Z1 

2  - 

-1  -1 

0  0 

TABLEAU  XVII 

TABLEAU  XVIII 

W4  Z2  Z3 

W1 

W4  w2 

z3  w: 

Z4 

2 

0  0-1 

-1 

Z4 

2 

0  0 

-1  -1 

W2 

-1 

0 

© 

LO 

0 

z2 

1/4 

0  -1 

-3/4  0 

W3 

2 

0-3  0 

0 

w3 

5/4 

0  0 

4/4  0 

Z1 

2  ■ 

-1  -1  0 

0 

Z1 

7/4  • 

-1  0 

3/4  0 
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4.  Comparison  Both  of  the  algorithms  described  in  sections  2  and  3  are 
adjacent  extreme  point  methods.  That  is,  each  algorithm  starts  at  an  ex¬ 
treme  point  of  a  closed  convex  set  and  proceeds  systematically  by  an 
exchange  of  basis  vectors  to  examine  an  adjacent  extreme  point. 

If  we  use  the  simplex  algorithm  in  Wolfe's  technique  then  we  see 
that  each  iteration  involves  a  pivot  operation.  However,  the  matrix  on 
which  this  operation  is  performed  is  at  least  (n+m)x2(n+m)^  in  the  case 
of  Wolfe's  algorithm  and  (n+m)x(n+ra)  in  the  principle  pivot  algorithm. 

This  means  four  times,  or  more,  as  many  operations  for  Wolfe's  method 
per  iteration  and  at  least  twice  the  storage  requirement.  The  revised 
simplex  method  offers  some  improvement  in  Wolfe's  algorithm  but  it  is  not 
significant  enough  to  balance  the  scales  in  the  direction  of  Wolfe's 
method. 

We  have  already  observed  that  both  algorithms  converge  under  the 

same  circumstances.  This  is  due  primarily  to  the  use  of  the  Kuhn-Tucker 

conditions  in  establishing  the  problem.  However,  the  semidefinite  case 

may  have  a.i  unbounded  solution.  The  principle  pivot  method  with  Cottle's 

modificaticn  detects  this  situation.  Wolfe's  algorithm  with  Charnes ' 

modificati jn,  however  changes  the  problem  itself.  It  will  converge  even 

though  the  solution  is  unbounded.  As  an  example,  consider 
2 

min  x|  -  2x^  -  3x2 , 
subject  to 

x  >,  0. 

This  problem  has  an  unbounded  solution  at  (1,  X£) .  However,  by  adding  c>p 
we  have 

^  This  number  is  increased  by  n  using  the  method  described  in  this  paper 
for  choosing  an  initial  basic  feasible  solution.  Hadley  [4]  describes 
an  alternative  method  which  requires  only  2(n+m)  columns.  However,  it 
involves  finding  the  inverse  of  the  basis  matrix. 
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min  (1  +  ^  e)  x^  +  ex^  -  2x^  -  3x2» 
subject  to  x  >  0. 

Which  has  a  solution  at  (1/(1  +  |  t),  3/e). 

Clearly,  as  e  -*■  0  we  see  that  the  solution  is  unbounded.  The  problem 
of  course  is  that  we  may  not  recognize  it  as  unbounded  in  practice  if  e  is 
too  large.  (Because  of  the  capacity  of  a  computer  for  example.)  With  care 
this  disadvantage  can  be  overlooked. 

One  interesting  difference  between  the  two  methods  is  that  in  Wolfe's 
method  the  solution  at  each  iteration  is  feasible.  However,  in  the  principle 
pivot  method  the  solution  is  not  feasible  until  convergence.  This  is  not 
very  important  since  it  would  be  useful  only  if  one  stopped  the  iterations 
before  optimality  was  reached.  Because  of  the  relative  speeds,  this  is  more 
likely  to  happen  with  Wolfe's  algorithm  anyway. 

We  have  seen  that  Wolfe's  method  requires  at  least  four  times  as  many 
operations  per  iteration  as  the  principle  pivot  method.  The  principle  pivot 
method  should  require  no  more  than  (n+m)  iterations  while  Wolfe's  will 
generally  require  from  (n+m)  to  2(n+m)  iterations.  From  this  observation, 
one  easily  concludes  that  the  principle  pivot  method  is  faster  than  Wolfe's 
and  hence  superior  since  it  solves  the  same  class  of  problems  as  Wolfe's. 
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